Thank you for your contribution @james-gould! We will review the pull request and get back to you soon.

@microsoft-github-policy-service agree

API change check

API changes are not detected in this pull request.

Build failures are due to the request and recorded URL being different (which is the change), please advise on next steps?

Hi @james-gould. Thank you for your contribution and interest in improving the Azure developer experience. The test failures do look to be related to test recordings and would require re-recording. Steps for doing so can be found in the recording section of the test framework README.

@jorgerangel-msft: I'm going to kick the live tests. Can you please take a look at the results and advise whether there's a gap in live testing coverage or in the scenario that is causing a 404 for @james-gould?

/azp run net - keyvault - tests

Sure, will wait for the tests to run. Thanks for getting back so quickly!

Will take a little time to record the tests again, need to set everything up locally. Will update the PR when it's ready!

I'm at a complete loss with this now, I've spent ~7 hours getting the test-proxy working, recording (not) working etc. Towards the end of last night I suddenly started getting 403 errors, despite the RBAC Key Vault Administrator role on a manually created and script created (via New-TestResources.ps1) vaults. I've created an entirely new Azure account + subscription, configured entra from scratch and still getting the same.

I'm gonna spend another 30-35 minutes trying to get these recordings sorted but I really can't spend much more time banging my head against the Azure wall trying to get all of this running unfortunately.

2 hours later still the same. Purged everything (again) and went back to scratch.

Set the values (removed for safety below, they're filled in locally) for these and wrote them into PS7:

${env:KEYVAULT_TENANT_ID} = ''
${env:KEYVAULT_CLIENT_ID} = ''
${env:KEYVAULT_CLIENT_SECRET} = ''
${env:KEYVAULT_SUBSCRIPTION_ID} = ''
${env:KEYVAULT_RESOURCE_GROUP} = ''
${env:KEYVAULT_LOCATION} = ''
${env:KEYVAULT_SKU} = ''
${env:AZURE_KEYVAULT_URL} = ''

Ran:

.\New-TestResources.ps1 -BaseName 'sdkresources' -ServiceDirectory 'keyvault' -SubscriptionId '<id>' -ResourceGroupName 'rg-sdkresources' -Location 'uksouth'

Which created the resources correctly:

Checking IAM for the vault:

Verified the .env is available in the keyvault directory, then ran:

setx KEYVAULT_TENANT_ID ${env:KEYVAULT_TENANT_ID}
setx KEYVAULT_CLIENT_ID ${env:KEYVAULT_CLIENT_ID}
setx KEYVAULT_CLIENT_SECRET ${env:KEYVAULT_CLIENT_SECRET}
setx KEYVAULT_SUBSCRIPTION_ID ${env:KEYVAULT_SUBSCRIPTION_ID}
setx KEYVAULT_RESOURCE_GROUP ${env:KEYVAULT_RESOURCE_GROUP}
setx KEYVAULT_LOCATION ${env:KEYVAULT_LOCATION}
setx KEYVAULT_SKU ${env:KEYVAULT_SKU}
setx AZURE_KEYVAULT_URL ${env:AZURE_KEYVAULT_URL}

Verified the values were set and persisted, then opened visual studio to run the tests. Ran a single one and got the prompt:

The resources needed to run the live tests could not be located.
Would you like to run the resource creation script? [y/n]:

Inputting Y then runs the New-TestResources.ps1 script but creates the resource group rg-jameskeyvault and then fails with the error in activity log:

[
  {
    "code": "StorageAccountAlreadyTaken",
    "target": "t5a619685e07b15a6prim",
    "message": "The storage account named t5a619685e07b15a6prim is already taken."
  }
]

but the resource group is empty. Inputting "n" just immediately fails the test with a 403.

I can't spend anymore time on this, I fix an error in one area and something else crops up. These resource scaffolding errors only started happening this morning, prior to that it was the 403 with:

    Azure.RequestFailedException : Caller is not authorized to perform action on resource.

I'm assuming that's related to this section, ie the credential isn't working but the .env isn't being used in the root of the keyvault directory.

@jsquire If you (or someone else) can help to resolve the config problems with me I'm happy to rerecord the tests and update the PR but if not I'll mark it as abandoned and put a workaround in the emulator.

Hi @james-gould, I'm looking at your issues with the recorder, my guess here is that a Key Vault HSM is required for some tests and that is why you are getting prompted with "The resources needed to run the live tests could not be located." message.

For that, you'll have to run a script command like this:

.\New-TestResources.ps1 -ServiceDirectory keyvault -AdditionalParameters @{enableHsm=$true; hsmLocation='uksouth'} -Location 'uksouth' -ResourceGroupName 'rg-sdkresources' -BaseName 'sdkresources' -UserAuth -SubscriptionId '<id>'

I'm going to try to replicate your issue in a new environment, I'll let you know what I find.

Thanks @JonathanCrd that makes sense, looking forward to see if it's repeatable your end. Assuming all is well, is there a way to record all tests rather than capturing a specific one with the x-recording-file key?

@james-gould When you run the tests, by default they will run in Playback mode, and if a test fails the test proxy tool will attempt to re-record it. It would look something like this.

If you want to record all the tests, you can do so by setting the Test Mote to Record and then run all of them, but in theory you wouldn't need to do that.

I was able to re-record after deploying the test resources with a managed HSM, so that was likely the problem. I'm going to push a commit to this PR to update the recordings.

Thank you again for your contribution

When you run the tests, by default they will run in Playback mode, and if a test fails the test proxy tool will attempt to re-record it. It would look something like this.

Annoying, I’d set it to record mode (uncommenting the enum in the base classes for the 2 clients) but couldn’t get the actual tests to proxy through. Using an API tool to speak to the proxy was fine.

Appreciate the info, I’ll rerun it my end tomorrow just to see it work for myself. Found a few weird things while working with the client at a granular level, would be useful experience for me. Appreciate you all taking the time!